Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: JeffW Date: Oct 24, 2002 12:10am
Forum: web Subject: crawling in 2002?

Has the updating of the archive slowed? Most of my site haven't been rearchived at all in 2002, though the sites have definitely had many changes in 2002.

My sites (examples:,,,, were archived in 2001 and previous years, but 2002 is blank.

What's going on? I don't think I've set anything in robots.txt that would stop the IA crawling. Other sites than mine that I've checked also seem to have few updates this year.

BTW, some of the faq links are broken.

Reply to this post
Reply [edit]

Poster: brewster Date: Oct 24, 2002 2:21am
Forum: web Subject: Yes we are crawling in 2002

Crawling is continuing by Alexa Internet (thank you to Alexa!), and the Internet Archive gets it with a 6 month time delay. But, as you observe we are behind on this. We are rebuilding indices so hopefully in the next couple of weeks we will be up to date (meaning only 6 months out of date).


Reply to this post
Reply [edit]

Poster: JeffW Date: Oct 24, 2002 2:35am
Forum: web Subject: Re: Yes we are crawling in 2002

good to know. You might post that info as a FAQ so people know that, at best, there is a 6 month delay.

btw, your forum login is a little funky. I login and it says it failed. If I refresh pages though it says I'm in.